CS-194-26: Project 5a

Abhijay Bhatnagar

Part 1: Shoot the Pictures

For my images, I took a couple shots from my kitchen, standing from the same point but rotating the camera in place.

Afterwards, I used the cpselect tool to define the following correspondence points.

Part 2: Recover Homographies

From here, I need to define the Homographic transformation H s.t. $PH=P'$, where p and p' are my two kitchen images. We can solve for H by setting up a system of linear equations in the following form (source):

$$PH = \begin{bmatrix} -x_1 & -y_1 & -1 & 0 & 0 & 0 & x_1x_1' & y_1x_1' & x_1' \\ 0 & 0 & 0 & -x_1 & -y_1 & -1 & x_1y_1' & y_1y_1' & y_1' \\ -x_2 & -y_2 & -1 & 0 & 0 & 0 & x_2x_2' & y_2x_2' & x_2' \\ 0 & 0 & 0 & -x_2 & -y_2 & -1 & x_2y_2' & y_2y_2' & y_2' \\ -x_3 & -y_3 & -1 & 0 & 0 & 0 & x_3x_3' & y_3x_3' & x_3' \\ 0 & 0 & 0 & -x_3 & -y_3 & -1 & x_3y_3' & y_3y_3' & y_3' \\ -x_4 & -y_4 & -1 & 0 & 0 & 0 & x_4x_4' & y_4x_4' & x_4' \\ 0 & 0 & 0 & -x_4 & -y_4 & -1 & x_4y_4' & y_4y_4' & y_4' \\ &&&&...&&&&\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\ \end{bmatrix} \begin{bmatrix}h1 \\ h2 \\ h3 \\ h4 \\ h5 \\ h6 \\ h7 \\ h8 \\h9 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0\\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ ... \\ 1 \end{bmatrix}$$

,

where $P$ is an $(2n + 1) \times 9$ matrix and $H$ is an $9x1$ matrix. The solution will give us the $8$ unknowns of $H$, with the last term corresponding to a $1$.

From here, we can input our corresponding points to compute P and then solve the system of equations to find the following transformation:

           H = array([[0.73, -0.02, 902.24],
                      [-0.07, 0.88, 184.59],
                      [-0.00, -0.00, 1.00]])

Part 3: Warp the Images

From our computed transform matrix H, we must use similar techniques as Project 3 to warp the right image to our desired homography (to mosiac with the left image). To do this, we will compute an inverse warp using the inverse of H, and interpreting points from the warping target points to the original image.

Original (left), Warped (Right)

Part 4: Image Rectification

Before we rectify our image, we first test our images on a few planar images to straighten them and see if our homographic transformation and warp work as expected. I found these images online (keyboard). Original points in red, projected rectangles in cyan.

<matplotlib.image.AxesImage at 0x1357b49d0>

As we can see, we are able to rectify the keyboard. We can try to recity another image for demonstration purposes. (source).

<matplotlib.image.AxesImage at 0x1368bd050>

Part 5: Blend the images into a mosaic

Now that we've written warping code and tested it, we can try to blend the original warped kitchen photos. I'll be blending the original left photo with the now warped right photo:

Left Image and Warped Right Image

<matplotlib.image.AxesImage at 0x1366813d0>

Now we are going to try this entire process on two other images from my .

<matplotlib.image.AxesImage at 0x150d0dc90>
<matplotlib.image.AxesImage at 0x152f8cd90>

Tell us what you've learned

This part of the project was incredible difficult for me. The underlying mathemathics and linear algebra transformation logic felt fairly straight forward. However, the details of image blending and alignment were incredible confusing, and took tons of research and trial and error to eventually accomplish. I have learned just how much processing goes into the simple panorama's our phones produce, and I've gained an appreciation for the complexity of an effective and efficient solution to what is fundamentally straightforward mathematics.

Note: I have probably spent more hours on Part A of this project than any other project to date...

proj5b

CS-194-26: Project 5b

Abhijay Bhatnagar

Part 1: Harris Corners

We will start with the same kitchen image as Part A, included again for reference.

For these two images, I used get_harris_corners with the corner_peaks tweak to find the harris points.

Part 2: Adaptive Non-Maximal Suppression

Here we will implement ANMS as described in this paper, suppressing our corners points to the best 300. For comparison, we are also including a graph of the 300 points by corner strength. We can see that our ANMS implementation sufficiently spreads out the interest points.

Part 3: Feature Descriptor Extraction

Following the procedure described in the paper, we will implement a feature descriptor extractor for the interest points as a precursor step to feature matching. For demonstration, I've selected a few sample points. For each sample point, I've shown the 40x40 pixel area the feature is extracted from, as well as the downsampled and adjusted extracted feature.

<Figure size 1440x720 with 0 Axes>

Part 4: Feature Matching

Now we are going to use our feature extractor and follow the paper's procedure to implement feature matching between the two images. We are using the Lowe method of thresholding set at $0.4$. The results are shown below.

As we can see, the far majority of the points are correctly matched. There exist a few false matches on visually similar features (such as edges of cupboards), but the far majority look to be correct.

Part 5: RANSAC

From here, we will use the RANSAC procedure from class to probabilistically select a good homographic transform H. This involves randomly selecting four points, computing the transform of those points, and measuring the distance between the actual corresponding point and the transform to see if it successfully mapped, within a threshold (I selected $\epsilon=5$ by experimentation). We run a 1000 iterations of this, and select the transform that correctly maps the greatest number of points.

I've included a H result from the RANSAC procedure, as well as the kitchen element blended by combining RANSAC with the full process above.

H = [[1.28 -0.10 -887.41]
 [0.06 1.15 -212.94]
 [0.00 -0.00 1.00]]

For demonstration purposes, I am showing the automatically generated mosiacs for the other 2 images included in part 1.

We've also included a new image to try it out on new data.

Tell Us What You Have Learned

This part of the project was a lot more straightforward, and it was a great exercise to follow the algorithm as described in the paper.